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Executive  Summary 

A  usability  evaluation  of  SLEP  (Service  Life  Extension  Plan)  VELCAC  (Virtual  Environment  Landing  Craft  Air 
Cushion)  was  undertaken  at  IFE II  in  San  Antonio,  TX  in  mid  February  of  2003.  This  evaluation  focused  on  a 
SLEP  VELCAC  system  that  is  currently  under  development,  but  had  a  functional  Craftmaster/Operator  station 
and  a  partially  functional  Engineer  station;  the  Navigator’s  station  was  not  interactive  at  the  time  of  evaluation. 
The  SLEP  VELCAC  system  is  designed  to  provide  differences  training  for  those  certified  LCAC  (Landing  Craft 
Air  Cushion)  crewmembers  that  are  transitioning  from  the  traditional  LCAC  to  the  SLEP  LCAC.  The  system 
is  also  designed  to  allow  mission  rehearsal  or  practice  flights  to  work  on  crew  coordination,  cockpit 
familiarization,  rehearsal  of  select  emergency  procedures,  and  practice  with  craft  features  unique  to  the 
SLEP  upgrade. 

The  usability  assessment  consisted  of  heuristic  evaluations  conducted  by  two  usability  engineers  and 
user  testing  of  SLEP  VELCAC’s  Craftmaster/Operator  and  Engineer  stations  to  the  extent  afforded  by  the 
system’s  maturity.  The  heuristic  evaluations  employed  traditional  usability  heuristics  (Nielsen,  1993)  and 
those  developed  specifically  for  virtual  environments  (Stanney,  Reeves,  Mollaghesmi,  Breaux,  &  Graeber, 
in  press)  to  find  positive  aspects  of  the  system’s  design  that  uphold  these  guidelines,  as  well  as  areas 
that  do  not  fully  adhere  to  accepted  design  standards.  One  SLEP  LCAC  Craftmaster/Operator  and  one 
SLEP  LCAC  Engineer  completed  user  testing  sessions  for  their  respective  cockpit  crew  positions.  User 
testing  sessions  focused  on  usability  issues  identified  by  the  heuristic  evaluations  and  interaction  with  the 
implemented  features  of  SLEP  VELCAC’s  Craftmaster/Operator  and  Engineer  crew  member  positions. 
These  sessions  took  the  form  of  modified  facilitated  free  play  (Stanney  &  Reeves,  1 995),  where  the 
participant  was  guided  by  the  usability  engineer  to  perform  certain  tasks  while  also  given  free  reign  to 
explore  the  system’s  capabilities  and  features.  Example  tasks  participants  were  asked  to  complete 
include  piloting  the  craft  between  two  waypoints  by  the  Craftmaster/  Operator,  as  well  as  recognizing  and 
responding  to  the  loss  of  APU  and  completion  of  a  fuel  transfer  in  response  to  a  craft  malfunction  by  the 
Engineer.  The  findings  from  both  the  heuristic  evaluations  and  user  testing  were  used  to  establish 
usability  testing  dependent  measures  and  their  associated  acceptability  criteria  for  future  usability  testing 
of  SLEP  VELCAC  when  more  mature  iterations  of  the  system  are  developed.  The  aforementioned 
dependent  measures  and  associated  criteria  levels  were  developed  with  input  from  the  usability 
engineers,  end-users,  system  developers,  and  sponsoring  agency  (ONR).  These  measures  are 
presented  in  the  form  of  a  usability  specification  matrix  (see  Table  5). 

The  findings  from  the  heuristic  and  user  testing  evaluations  revealed  a  variety  of  positive  aspects  and 
user  responses.  Most  notably,  users  stated  that  they  could  foresee  the  utility  of  SLEP  VELCAC  to  support 
differences  training  when  transitioning  from  the  traditional  to  the  SLEP  LCAC.  In  addition,  the 
Craftmaster/Operator  noted  that  the  virtual  craft’s  dynamics  and  handling  were  replicated  in  a  manner 
consistent  with  the  actual  craft.  Results  from  the  heuristic  evaluations  indicated  SLEP  VELCAC  does  a 
good  job  of  replicating  the  crewstations  and  has  the  beginnings  of  a  well  designed  training  system.  These 
analyses  and  the  associated  user  testing  revealed  a  number  of  usability  concerns  of  varying  priority 
levels  that,  if  successfully  resolved,  have  the  potential  to  substantially  improve  the  usability  of  the  SLEP 
VELCAC.  Table  1  presents  high  priority  usability  concerns  and  associated  potential  solutions.  A  more 
inclusive  table  of  usability  concerns  is  presented  in  Table  4. 
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Table  1 .  High  priority  usability  concerns  for  SLEP  VELCAC  (February,  2003). 


Usability  Problem 

User 

Impact 

Recommended  Solution 

Unnatural  to  zoom  via  repeatedly 
depressing  keys;  can  cause  user 
disorientation/ frustration 

H 

•  Add  quick  zoom  feature,  possibly  by  adding  default  values  matching 
how  much  user  would  zoom  in  to  be  able  to  view  a  screen  (e.g.,  a  user 
may  not  need  a  small  incremental  change,  rather  a  2X  zoom) 

Changing  point  of  view  via  repeatedly 
depressing  keys  is  unnatural;  can 
cause  user  disorientation/  frustration 

H 

•  Add  a  mouse  function  that  allows  the  mouse  to  act  as  the  user* s  eyes 
(for  example,  if  the  user  scrolls  to  the  left,  the  point  of  view  moves  to 
the  left) 

Unnatural  flow  of  information 
gathering/  integration  resulting  from 
having  to  zoom  in  on  main  or  auxiliary 
display,  which  excludes  supplemental 
information  provided  on  other  screen 

H 

•  Change  the  default  view  to  increase  clarity  of  the  three  screens 

necessary  for  Engineer’s  task  performance.  To  accomplish  this, 
consider  removing  non-task  relevant  graphics  on  the  default  view 
screen,  changing  font  sizes,  and  increasing  contrast  of  each  screen 

Users’  memory  taxed  with  having  to 
leam/remember  keyboard  commands 

H 

•  Use  mouse  or  touch  screen  functionality  to  replace  keyboard  function 
(or  use  the  keyboard  as  a  redundant  backup) 

Users’  memory  load  is  taxed  by 
necessity  to  zoom  in  on  the  main  or 
auxiliary  display  to  read  information, 
which  excludes  ability  to  see  the 
other  display 

H 

•  Change  the  default  view  to  increase  clarity  of  the  three  screens 

necessary  for  Engineer’s  task  performance.  To  accomplish  this, 
consider  removing  non-task  relevant  graphics  on  the  default  view 
screen,  changing  font  sizes,  and  increasing  contrast  of  each  screen 

Labeling  of  displays  is  hard  to  read 
thus  requiring  user  to  recall 
functionality  accessed  via  controls 

H 

•  Increase  clarity  of  alphanumeric  text  via  font  size,  sharpness,  and 
contrast 

Inconsistent  functioning  of  “ack”  key 
to  acknowledge  alerts/  alarms 

H 

•  Change  functionality  to  match  operational  functionality 

No  indication  of  bow  thruster  status 
(stow  v.  operate)  or  direction  (forward 
v.  reverse)  without  viewing  synthetic 
HUD,  which  does  not  exist  in  the  craft 

H 

•  Create  a  bow  thruster  switch  on  the  yoke  that  replicates  the 
functionality  of  the  bow  thruster  switch  in  the  SLEP  LCAC 

No  ability  to  undo  errors,  thereby 
avoiding  more  critical  errors 

H 

•  Add  an  undo  option,  in  which  a  user  can  undo  the  last  input  (e.g.  if  a 
user  accidentally  turns  off  an  engine,  etc.) 

•  Add  qo  back  function  so  users  can  pick  up  situation  at  earlier  point 

No  on-line  help  available 

H 

•  Create  a  brief  tutorial  of  how  to  interact  with  the  VE 

•  Create  a  cut  out  keyboard  cheat  sheet  to  illustrate  functionality 

•  Add  a  menu  capability  that  allows  users  to  have  access  to  help 

Engineering  station  does  not  allow 
interaction  with  buttons  on  overhead 
console 

H 

•  Make  high  priority  buttons,  switches,  etc.  fully  functional  (see  Stanney, 
Graeber,  and  Milham,  2002) 

Text  on  main  and  auxiliary  displays  is 
illegible  when  zoomed  out  and 
occasionally  illegible  when  zoomed  in 

H 

•  Ensure  test  is  legible  at  both  default  and  zoom  in  displays.  Character 
height  for  maximum  legibility/  readability  is  20-22  arc  min,  for  legibility 
minimum  acceptable  is  16  arc  min;  if  individual  characters  do  not  have 
to  be  read,  10  arc  min  is  minimum  acceptable.  Use  san  serif  font  for 
small  text  and  low  resolution  displays,  otherwise  use  serif  fonts 
rSource:  DOT/FAA/CT-96/01]’ 

Users  did  not  realize  that  the  upper 
panels  for  each  crewmember  position 
were  represented 

H 

•  Create  an  introductory  screen  illustrating  the  active  parts  of  the  display 

Repeated  system  lockups  required 
rebooting  of  the  system 

H 

•  Minimize  system  crashes 

When  flying  over  land  the  engineer’s 
controls  became  inactive 

H 

•  Implement  fix 

Zoom  in  on  engineer’s  main  display 
(i.e.  the  4  button)  was  not  completely 
functional,  it  needed  to  be  depressed 

H 

•  Implement  fix 

numerous  times  (in  some  cases  15 


lifs  function 

The  inactive  keys  are  causing 
confusion  because  they  are  in  the 
incorrect  non-default  condition 


Change  default  conditions  to  match  real  world  default 


Key:  H=High;  M-H=Medium-High 

1  To  account  for  both  size  of  symbols/characters  and  viewing  distance,  visual  angle  is  used  as  unit  of  measurement.  Visual  angles  are 
specified  in  terms  of  minutes  of  arc  or  degree  (1  degree  =  60  minutes  of  arc). 
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Introduction 


Full  Product  Description 

The  system  under  evaluation  is  the  version  of  SLEP  (Service  Life  Extension  Plan)  VELCAC  (Virtual 
Environment  Landing  Craft  Air  Cushion)  demonstrated  at  IFE II  in  San  Antonio,  TX  mid  February  2003. 
This  system  is  currently  undergoing  an  iterative  development  process  in  an  effort  to  create  a  virtual 
environment  training  apparatus  that  supports  both  transition  from  the  traditional  LCAC  (Landing  Craft  Air 
Cushion)  to  the  SLEP  LCAC  and  mission  rehearsal  or  practice  flights  to  work  on  crew  coordination, 
cockpit  familiarization,  rehearsal  of  select  emergency  procedures,  and  practice  with  craft  features  unique 
to  the  SLEP  upgrade.  The  components  of  SLEP  VELCAC  that  were  evaluated  included  the 
Craftmaster/Operator’s  station  and  the  Engineer’s  station;  the  Navigator’s  station  was  not  evaluated  due 
to  its  limited  development  at  the  time  of  assessment.  It  should  be  noted  that  the  Craftmaster/Operator 
position  and  Engineer  position  were  not  fully  developed  and  thus  bounded  the  depth  of  the  usability 
evaluation.  The  Craftmaster/Operator  station  had  functional  craft  controls  and  related  gauges/displays  for 
flight,  however,  the  auxiliary  display  that  allows  viewing  of  navigation  or  engineering  related  information 
was  not  functional.  The  Engineer’s  station  had  more  limited  development,  with  the  functionality  of  the 
UKB  (Universal  Keyboard),  main  display,  and  auxiliary  display  curtailed  to  support  immediate  actions 
associated  with  specific  casualties  (i.e.  loss  of  APU)  and  monitoring  of  the  plant  during  normal  operations. 

The  intended  user  population  for  SLEP  VELCAC  comprises  certified  LCAC  crewmembers  that  are 
transitioning  from  the  traditional  LCAC  to  the  SLEP  LCAC,  as  well  as  SLEP  LCAC  certified 
crewmembers.  In  essence,  the  user  population  consists  of  expert  LCAC  crewmembers  that  require  a 
means  for  familiarization  with  the  SLEP  upgrades  and  ability  to  enhance  crew  fluidity  that  may  have  been 
altered  by  SLEP’s  technological  advancements.  For  a  thorough  description  of  the  user  population’s 
characteristics  the  reader  is  referred  to  Stanney,  Graeber,  and  Milham  (2002a;  2002b).  The  intended 
environment  for  use  of  SLEP  VELCAC  is  both  as  a  school  house  training  system  and  a  deployable 
system  that  could  be  utilized  dockside  or  underway.  A  summary  of  SLEP  LCAC  crewmembers’  views  on 
intended  use  and  after  action  review  requirements  is  presented  in  Appendix  A. 


Test  Objectives 

The  objective  of  this  usability  evaluation  was  the  development  of  a  usability  specification  matrix  via  heuristic 
evaluation  and  user  testing.  In  achieving  this  main  objective,  sub-objectives  were  also  attained,  which  included 
cataloging  of  positive  design  features  and  usability  concerns  associated  with  the  SLEP  VELCAC,  as 
implemented  at  the  time  of  evaluation.  The  usability  specification  matrix  (USM)  transforms  general  usability 
objectives  (i.e.,  efficiency,  intuitiveness,  satisfaction)  into  specific  measures  that  constitute  usability 
requirements  for  a  given  system  (Wixon  &  Wilson,  1997).  The  USM  provides  a  means  for  evaluating  SLEP 
VELCAC  in  later  iterations  of  its  design  by  establishing  usability  criteria,  as  well  as  acceptable  levels  for  each 
criterion.  The  USM  included  herein  utilizes  four  levels  of  acceptability:  unacceptable,  minimum,  planned,  and 
best  case.  The  unacceptable  level  denotes  a  value  for  an  attribute  that  signifies  a  serious  usability  violation  that 
must  be  mitigated  via  redesign  to  avoid  catastrophic  (i.e.,  unrecoverable)  usability  impairment  of  the  system. 
The  minimum  level  establishes  the  minimum  acceptable  performance  for  the  attribute  and  indicates  that 
redesign  should  be  undertaken  to  improve  usability.  The  planned  level  is  considered  to  be  the  target  for 
usability  success  indicating  that  little  to  no  redesign  is  required  for  assuring  a  usable  system.  Finally,  the  best 
case  level  is  a  level  of  performance  that  is  optimal  and  could  be  theoretically  achieved  if  a  concerted  effort,  void 
of  budget  and  time  constraints,  was  devoted  to  system  design,  development,  and  usability  engineering.  This 
level  is  specified  primarily  as  a  target  for  future  design  iterations. 

To  compile  the  attributes  for  a  usability  specification  matrix  various  approaches  can  be  used,  in  this  case 
heuristic  evaluation  implementing  general  usability  heuristics  (Nielsen,  1993)  and  virtual  environment  specific 
heuristics  (Stanney,  et  al.,  under  review),  as  well  as  user  testing  were  used.  These  approaches  reveal  usability 
concerns  associated  with  the  system  that  can  be  folded  into  the  usability  specification  matrix  as  attributes  to  be 
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tracked  over  the  development  of  the  system.  Typically,  only  usability  issues  of  high  concern  are  included  in  the 
usability  specification  matrix  because  less  serious  usability  violations  can  often  be  remedied  through  minimal 
redesign  efforts.  Specific  usability  questions  asked  by  the  development  team  are  addressed  in  Appendix  B. 


Usability  Attribute  Table 

Before  constructing  the  usability  specification  matrix  an  initial  step  was  the  development  of  a  usability  attribute 
table  based  on  the  intended  use  of  the  system.  The  usability  attribute  table  incorporates  a  variety  of  aspects 
that  may  affect  the  usability  of  the  system  and  provides  a  focus  for  evaluative  efforts  and  the  generation  of  the 
usability  specification  matrix.  In  the  case  of  SLEP  VELCAC  information  pertaining  to  the  users,  tasks,  system, 
and  environment  was  captured  in  previous  data  collection  efforts  (see  Stanney  et  al.,  2002a;  2002b)  and  fused 
to  create  a  compilation  of  key  usability  attributes  for  SLEP  VELCAC.  These  attributes  are  presented  in 
Appendix  C  and  serve  as  a  basis  for  guiding  data  collection  efforts,  generating  the  usability  specification  matrix, 
and  specifying  usability  metrics.  With  the  aforementioned  usability  attribute  table  in  place,  evaluative  efforts  of 
SLEP  VELCAC  could  commence  as  described  in  the  Method  section. 


Method 

Experimental  Design 

The  experimental  design  for  this  evaluation  consisted  of  two  phases.  The  first  phase  was  the  heuristic 
evaluation  where  general  usability  rules  of  thumb  (Nielsen,  1993)  and  virtual  environment  specific  usability 
guidelines  (Stanney  et  al.,  in  press)  were  used  by  usability  engineers  to  systematically  evaluate  the  SLEP 
VELCAC  system.  The  second  phase  consisted  of  user  testing  designed  to  validate  the  heuristic  evaluation 
findings  and  discover  additional  usability  concerns  via  participants  completing  tasks  and  a  free  play  session. 
Specific  metrics  were  not  employed  for  the  user  testing  portion  due  to  the  immaturity  of  the  system;  instead  the 
goal  of  the  two  phases  was  to  establish  which  metrics  should  be  utilized  in  subsequent  usability  evaluations, 
once  the  system  is  more  mature. 

Test  Facility 

The  evaluation  was  conducted  at  SWRI  (Southwest  Research  Institute)  in  San  Antonio,  TX.  The  SLEP 
VELCAC  system  was  setup  in  a  lab  space  that  served  the  purpose  of  a  development  and  demonstration  facility 
for  a  variety  of  virtual  environment  systems.  The  setting  for  testing  was  akin  to  potential  scenarios  for  SLEP 
VELCAC’s  use  (i.e.,  classroom,  deployed)  due  to  the  volume  of  people  and  other  systems  actively  working  in 
the  room  during  data  collection.  This  created  a  relatively  noisy  and  dynamic  environment  where  interruptions 
from  individuals  not  involved  with  data  collection  occasionally  occurred.  The  usability  engineers  conducting  the 
user  testing  session  do  not  feel  these  conditions  withdrew  from  the  validity  of  findings,  but  instead  enhanced 
them  because  the  setting  was  akin  to  actual  training  conditions  at  an  ACU  (Assault  Craft  Unit). 

Participants 

For  the  user  testing  portion  of  the  evaluation  two  participants  were  involved.  Both  participants  were  certified 
SLEP  LCAC  crewmembers  from  ACU-5  at  Camp  Pendleton,  CA;  one  a  Craftmaster/Operator  and  the  other  an 
Engineer.  These  participants  were  selected  because  they  are  among  the  few  individuals  that  have  completed 
SLEP  LCAC  differences  training  and  logged  flight  hours  in  a  SLEP  LCAC.  User  profile  data  (see  Appendix  D  for 
questionnaire)  were  collected  on  these  participants  as  well  as  a  certified  SLEP  LCAC  Navigator  who  gave  input 
on  the  SLEP  VELCAC  system,  their  data  are  as  follows.  These  crewmembers  have  logged  between  200-330hr 
in  a  NDI  equipped  LCAC,  800-1 400hr  in  a  non-NDI  equipped  LCAC,  and  3-40hr  in  a  SLEP  LCAC.  All  three 
crewmembers  will  undertake  the  role  of  SLEP  LCAC  instructor  as  more  personnel  complete  the  SLEP  LCAC 
differences  training.  The  survey  respondents  felt  that  there  was  a  high  level  of  automation  in  their  jobs,  they 
enjoy  working  with  computers,  and  felt  that  the  automation  provided  in  SLEP  LCAC  has  made  their  job  easier. 
These  individuals  also  noted  that  they  find  it  challenging  and  rewarding  to  learn  new  computer  applications,  and 
that  learning  these  applications  pays  off  because  it  helps  them  complete  a  task  faster  or  perform  their  job  better. 
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These  participants  have  been  flying  LCACs  for  7-8  years  and  consider  themselves  experts  in  a  traditional 
LCAC,  but  only  novice  or  experienced  in  the  SLEP  LCAC.  Their  previous  experience  with  U.S.  Navy  training 
systems  resulted  in  the  impression  that  these  systems  were  somewhat  effective  in  training  critical  job  skills  and 
somewhat  easy  to  use.  Finally,  all  three  respondents  were  male  between  the  ages  of  26-40,  possessed 
accurate  depth  perception,  were  not  colorblind  or  physically  disabled  in  a  manner  that  would  require  special 
consideration,  and  one  required  the  use  of  glasses  to  correct  farsighted  vision. 

Procedure 

Presented  below  is  a  brief  discussion  of  the  heuristics  used  to  evaluate  SLEP  VELCAC  followed  by  the  tasks 
completed  by  participants  during  user  testing.  The  heuristics  employed  were  twofold,  one  was  a  more  general 
set  of  guidelines,  while  the  other  was  virtual  environment  specific.  As  a  result,  these  provide  a  broad,  high  level 
means  for  assessing  a  product;  an  operational  definition  for  each  is  provided  in  Table  2  below. 


Table  2.  General  usability 

heuristics  and  their  operational  definitions. 

Heuristic 

Operational  Definition 

Simple  and  Natural 
Presentation 

Task  relevant  information  presented  in  a  logical,  natural,  and  streamlined  fashion 

Speak  the  User’s 

Lanquage 

System  interaction  dialogue  that  is  clear,  concise,  and  consistent  with  user- 
defined  domain;  this  could  include  mappinq  to  the  user’s  conceptual  model 

Minimize  Memory  Load 

Recoanition  is  better  than  recall;  capitalize  on  affordances 

Consistency 

Interface  should  respond  to  user’s  actions  in  an  expected  and  reliable  manner 

Feedback 

Users  should  be  aware  of  what  system  is  doing  and  how  it  is  interpreting  their 
input 

Clear  Exits 

Users  should  never  feel  trapped  in  a  site,  mode,  or  state  of  a  system;  undo 
capability  is  essential 

Shortcuts 

Provide  expert  users  with  means  to  quickly  access  desired  system  states 

Errors,  Error  Handling, 

Error  Prevention 

Provide  error  correction  and  recovery  before  a  permanent  change  occurs;  tell 
users  when  and  how  a  mistake  was  made,  what  can  be  done  to  correct  it,  and 
how  to  avoid  the  mistake  in  the  future 

Help  1 

Provide  on-line  help  within  the  application;  provide  plans  or  maps  of  the  VE 

The  virtual  environment  specific  usability  heuristics  employed  in  this  evaluation  were  developed  by  Stanney,  et 
al.  (in  press)  to  provide  the  usability  engineering  community  a  set  of  guidelines  by  which  virtual  environments 
and  their  unique  usability  challenges  could  be  evaluated.  These  guidelines  go  beyond  traditional  usability 
considerations  taking  into  consideration  aspects  such  as  the  design  of  wayfinding  and  navigational  techniques, 
object  selection  and  manipulation,  visual,  auditory  and  haptic  system  outputs,  presence,  immersion,  and 
system  comfort,  as  well  as  minimizing  sickness  and  deleterious  aftereffects.  The  heuristics  from  Stanney  et  al.’s 
(in  press)  that  are  relevant  to  SLEP  VELCAC  are  provided  below  in  Table  3. 


Table  3.  Virtual  environment  specific  usability  heuristics  and  their  operational  definitions. 


Heuristic 

-  -  *  ■■  ■  *  ■  . .  1  . . 

Operational  Definition  _ 

Interaction 

Interaction  should  be  natural,  efficient,  and  appropriate  for  target  users,  domains,  and  task 
goals 

Wayfinding 

Provide  ability  to  maintain  knowledge  of  one’s  location  and  orientation  while  moving 
throughout  a  designed  space 

Intuitive  naviqational  control  should  be  provided  in  a  streamlined  fashion 

Object  selection 
and  manipulation 

Process  of  indicating  virtual  objects  within  an  environment  to  reposition,  reorient,  or  query 
them  should  not  be  awkward  or  disorienting 

Visual 

Consideration  of  effectiveness  of  stereoscopic  support,  spatial  resolution,  field-of-view- 
update  rates,  refresh  rates,  and  user  comfort  and  acceptance  should  be  given _ 

Engagement  in  virtual  environment  should  be  fostered  and  sustained,  thereby  enhancing 
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sense  of  presence _ _ _ 

Presence 

Enhance  the  subjective  perception  of  experiencing  oneself  as  being  in  a  computer- 
qenerated  environment  rather  than  in  one’s  actual  physical  location 

Immersion 

Enhance  the  perception  of  oneself  being  enveloped  by,  included  in,  and  interacting  with 
an  environment  that  provides  a  continuous  stream  of  stimuli  and  experiences 

Comfort 

Overall  physical  discomfort  should  be  minimized,  while  user  safety  is  maximized 

Note:  this  table  excludes  the  Stanney  et  al.’s  (in  press)  heuristics  on  haptic  and  auditory  output,  as  well 
as  sickness  and  aftereffects  because  it  is  not  felt  they  are  a  concern  at  this  juncture. 


The  user  testing  portion  of  the  usability  evaluation  consisted  of  the  Engineer  and  Craftmaster/Operator 
performing  scenario  setup  procedures,  and  general  navigation  and  object  manipulation  in  the  VE;  the  Engineer 
was  also  asked  to  complete  a  specified  casualty  procedure.  Due  to  the  current  state  of  SLEP  VELCAC’s 
development,  scenario  setup  was  limited  to  logging  into  the  system.  When  a  more  dynamic  scenario  setup 
capability  is  developed  the  task  of  configuring  and  beginning  a  user  session  will  be  revisited  more  thoroughly. 

The  next  area  of  focus  for  user  testing  completed  by  both  Engineer  and  Craftmaster/Operator  included  general 
navigation  through  the  VE;  manipulating  objects  in  the  VE;  examining  the  readability  and  utility  of  essential 
screens/displays/gauges,  and  UKB  buttons  and  auxiliary  displays.  (Note  only  the  Engineer  had  a  functioning 
UKB  and  auxiliary  display).  These  tasks  are  critical  to  the  user’s  ability  to  interact  with  and  within  the  VE,  as 
well  as  obtain  requisite  information  to  support  the  intent  of  a  user  session. 

Finally,  the  Engineer  completed  a  casualty  procedure  that  involved  mitigating  the  loss  of  the  APU  (see 
Appendix  E  for  an  updated  list  of  tasks  that  an  Engineer  performs  to  deal  with  loss  of  the  APU).  APU  failure  was 
chosen  as  a  representative  casualty  for  this  iteration  of  SLEP  VELCAC  because  it  is  a  realistic  malfunction  that 
the  Engineer  would  rehearse  in  both  SLEP  LCAC  differences  training  and  in  an  operational  setting.  However, 
only  portions  of  the  immediate  actions  associated  with  the  casualty  could  be  evaluated  due  to  the  maturity  of 
SLEP  VELCAC.  As  a  result,  user  testing  focused  on  those  portions  of  the  casualty  mitigation  procedure  that 
were  functional,  as  well  as  identifying  additional  key  elements  of  the  task  needed  to  round  out  the  complete 
suite  of  functionality  and  displays  requisite  for  effectively  undertaking  appropriate  immediate  actions. 

While  performing  the  aforementioned  tasks,  users  were  provided  with  a  “cheat  sheet”  of  keyboard  and  mouse 
functions  to  assist  in  choosing  the  proper  means  for  manipulating  objects  and  moving  within  the  environment. 
However,  it  should  be  noted  that  the  “cheat  sheet”  could  only  be  accessed  via  a  paper  based  document  and 
thus  presented  a  barrier  to  its  efficient  use  while  completing  tasks  with  SLEP  VELCAC.  Finally,  user  testing 
conducted  on  this  iteration  of  SLEP  VELCAC  was  focused  more  on  a  qualitative  evaluation  of  the  system  to 
identify  areas  of  concern  via  errors  and  user  frustration  rather  than  an  objective  cataloging  of  traditional  usability 
metrics  (i.e.,  time  in  error,  number  of  errors,  time  to  complete  tasks,  etc.). 


Results 

Problem-Solution  Table 

The  problem  solution  table  below  (see  Table  4)  presents  a  summary  of  the  usability  concerns  based  on  the 
heuristic  evaluation  and  user  testing  findings  for  both  the  general  and  virtual  environment  heuristics  (see 
Appendix  F  for  complete  heuristic  evaluation  findings).  Along  side  each  usability  concern,  the  table  provides 
potential  impact  on  users,  suggested  remedies,  and  the  importance  of  having  the  concern  addressed  via 
redesign.  This  table  is  based  on  the  limited  SLEP  VELCAC  functionality  available  today.  After  each  IFE 
usability  evaluation,  a  problem  solution  table  will  be  generated  that  focuses  on  advancing  functionality  as  the 
SLEP  VELCAC  system  matures. 
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Table  4.  Problem-solution  table  for  SLEP  VELCAC  usability  concerns. _ 

Usability  Concern  |  User  |  Recommended  Solution 


Simple  and  Natural  Presentation 
Heuristic  _ 


Using  the  mouse  to  interact  with  the  UKB  is 
not  natural  _ 


Using  the  mouse  to  interact  with  other 
switches,  knobs,  dials  is  not  natural 


Rotating  point  of  view  up  with  the  “S”  key 
and  down  with  the  ‘W’  key  is  counter 
intuitive;  can  cause  user  disorientation/ 
frustration 


Unnatural  to  zoom  via  repeatedly 
depressing  keys;  can  cause  user 
disorientation/  frustration 


Resolution 


Changing  point  of  view  via  repeatedly 
depressing  keys  is  unnatural;  can  cause 
user  disorientation/  frustration  _ 


Unnatural  flow  of  information  gathering/ 
integration  resulting  from  having  to  zoom  in 
on  main  or  auxiliary  display,  which  excludes 
supplemental  provided  on  other  screen 


Users  did  not  realize  that  the  upper  panels 
for  each  crewmember  position  were 
represented 


Users  suggested  horizon  shot  wasn  t 
correct,  too  much  water,  not  enough  sky 


Alarms  on  engineer’s  auxiliary  page  flash  in 
the  actual  craft  when  they  have  not  been 
acknowledged,  but  they  do  not  in  SLEP 
VELCAC  _ 


The  inactive  keys  are  causing  confusion 
because  they  are  in  the  incorrect  non-default 
condition 


k  the  User’s  Lanquaqe  Heuristic 


User’s  language  is  not  utilized  to  its  fullest 
extent  possible  in  the  -cheat  sheef 


Minimize  Memory  Load 


User's  memory  taxed  with  having  to  learn/ 
remember  keyboard  commands 


User's  memory  load  is  taxed  by  necessity  to 
zoom  in  on  the  main  or  auxiliary  display  to 
read  information,  which  excludes  ability  to 
see  the  other  display 


Labeling  of  displays  is  hard  to  read  thus 
requiring  user  to  recall  functionality 
accessed  via  controls 


Consister  r  Heuristic 


Inconsistent  functioning  of  the  “ack"  key  to 
acknowledge  alerts  and  alarms 


Add  touch  screen  capability  so  that  users  can 
interact  directly  with  the  UKB 


Allow  users  to  directly  touch  interface  via 
touchscreen 


Switch  the  functionality  of  the  keys:  change  the  “S” 
key  to  shift  point  of  view  down  and  the  “W"  key  to 
shift  point  of  view  up 


Add  quick  zoom  feature,  possibly  by  adding  default 
values  matching  how  much  a  user  would  zoom  in  to 
be  able  to  view  a  screen  (e.g..  a  user  may  not  need 
a  small  incremental  change,  rather  a  2X  zoom) 


Add  a  mouse  function  that  allows  the  mouse  to  act 
as  the  user’s  eyes  (for  example,  if  the  user  scrolls  to 
the  left,  the  point  of  view  moves  to  the  left) _ 


Change  the  default  view  to  increase  clarity  of  the 
three  screens  necessary  for  Engineer’s  task 
performance.  To  accomplish  this,  consider 
removing  non-task  relevant  graphics  on  the  default 
view  screen,  changing  font  sizes,  and  increasing 
contrast  of  each  screen 


Create  an  introductory  screen  illustrating  the  active 
parts  of  the  display 


Change  the  water/sky  ratio  to  reflect  real  world  visual 
(40%/60%) 


Changing  functionality  to  reflect  real  world 
functionality 


Change  default  conditions  to  match  real  world 
default 


Change  terms  to  be  more  coherent  (e.g.  zoom  in  vs. 
move  view  left,  right,  forward,  backward 


Use  mouse  or  touch  screen  functionality  to  replace 
keyboard  function  (or  use  the  keyboard  as  a 
redundant  backup) 


Change  the  default  view  to  increase  clarity  of  the 
three  screens  necessary  for  Engineer’s  task 
performance.  To  accomplish  this,  consider 
removing  non-task  relevant  graphics  on  the  default 
view  screen,  changing  font  sizes,  and  increasing 
contrast  of  each  screen 


Increase  clarity  of  alphanumeric  text  via  font  size, 
sharpness,  and  contrast 


Create  consistent  mapping  of  key  presses 


Change  functionality  to  match  operational 
functionali 


Visuals  do  not  give  any  sense  of  craft  speed 

M 

•  Add  additional  visual  cues  to  increase  optic  flow 

No  collision  detection  over  land 

M 

•  Add  collision  detection 

M-H  (H  if 
using  for 


System  locks  up  without  telling  the  user  why 
it  locked  up 


No  indication  of  bow  thruster  status  (stow  v. 
operate)  or  direction  (forward  v.  reverse) 
without  viewing  synthetic,  HUD  which  does 
not  exist  in  the  craft  _ 


Clear  Exits  Heuristic 


No  pause  capability _ 


No  ability  to  go  back  to  a  particular  part  of  a 
scenario  (can  only  restart) _ 


Add  diagnostic  error  messages  (e.g.  what  happened 
and  what  the  user  can  do  to  go  back  or  restart,  if 
necessary)  _ 


Create  a  bow  thruster  switch  on  the  yoke  that 
recreates  the  functionality  of  the  bow  thruster  switch 
in  the  SLEP  LCAC 


Add  a  pause  capabili 


Add  a  menu  system,  in  which  users  can  exit,  pause, 
and  qo  back  to  particular  parts  of  the  scenario _ 


Shortcuts  Heuristic 


No  ability  to  choose  where  along  a  route  or 
task  completion  to  start  a  scenario  _ 


Errors,  Error  Handling,  Error 
Prevention  Heuristic 


No  ability  to  undo  errors,  thereby  avoiding 
more  critical  errors 


When  system  locks  up  there  is  no  indication 
of  why  or  when  via  error  messages 


Repeated  system  lockups  required 
rebooting  of  the  system 


When  transitioning  from  surfzone  to  land  the 
Craftmaste r/Operator's  heading  indicator 
malfunctioned  _ 


When  flying  over  land  the  engineers 
controls  became  inactive  _ _ 


Zoom  in  on  engineer’s  main  display  (i.e.  the 
4  button)  was  not  completely  functional,  it 
needed  to  be  depressed  numerous  times  (in 
some  cases  15  times)  before  performing  it’s 
function 


Help  Heuristic 


No  on-line  help  available 


Add  a  menu  system,  in  which  users  can  start  at 
rticular  parts  of  the  scenario 


Add  an  undo  option,  in  which  a  user  can  undo  the 
last  input  (e.g.  if  a  user  accidentally  turns  off  an 
engine,  etc.) 

Add  go  back  function  so  users  can  pick  up  situation 
at  earlier  point 


Provide  appropriate  error  message  for  system  lock 
ups  _ 


Minimize  fatal  system  errors 


Implement  fix 


Implement  fix 


Add  a  brief  tutorial  on  how  to  interact  with  the  VE 
Add  a  cut  out  keyboard  cheat  sheet  to  illustrate 
functionality 

Add  a  menu  capability  that  allows  users  to  have 
access  to  help  _ 


Interaction  Heuristic 


Interaction  with  SLEP  menu  screens  limited 
toUKB  _ 


Engineering  station  does  not  allow 
interaction  with  buttons  on  overhead  console 


Navigation  Heuristic 


Lack  of  collision  detection  allows  user  to 
zoom  point  of  view  outside  of  cockpit 


No  solution  is  required 


Make  high  priority  buttons,  switches,  etc.  fully 
functional  (see  Stanney,  Graeber,  and  Milham, 
2002) 


Constrain  movement  of  viewpoint  to  within  the 
cockpit  _ 


Visual  Heuristic 


Text  on  main  and  auxiliary  displays  is 
illegible  when  zoomed  out  and  occasionally 
illegible  when  zoomed  in 


Ensure  text  is  legible  at  both  default  and  zoom  in 
displays.  Character  height  for  maximum  legibility/ 
readability  is  20-22  arc  min,  for  legibility  minimum 


acceptable  is  16  arc  min;  if  individual  characters  do 
not  have  to  be  read,  10  arc  min  is  minimum 
acceptable.  Use  san  serif  font  for  small  text  and  low 
resolution  displays,  otherwise  use  serif  fonts 
[Source:  DOT/FAA/CT-96/OH 

When  transitioning  from  surfzone  to  land  a 
qreenish  brown  bar  fills  a  majority  of  screen 

L 

•  Implement  fix 

L 

'Engineers  oveitiead  panels  need  to  be  active  if  the  intended  use  of  the  system  includes  start  up  procedures  and  formation  flying 


Usability  Specification  Matrix 

The  usability  specification  matrix  presented  below  in  Table  5  contains  traditional  usability  metrics  that  are 
applicable  to  a  variety  of  systems,  including  virtual  environments,  and  critical  usability  concerns  unveiled  in  the 
heuristic  evaluation  and  user  testing.  This  table  was  drafted  by  the  usability  engineers  conducting  the  SLEP 
VELCAC  evaluation  and  discussed  in  a  focus  group  that  invited  feedback  from  end  users,  system  developers, 
and  the  sponsoring  agency.  The  result  of  that  discussion  was  the  generation  of  values  for  the  various  levels  of 
acceptance  for  each  usability  attribute  listed  in  the  matrix;  in  some  cases  values  will  be  established  at  a  later 
date  when  additional  data  on  the  system  are  provided  by  system  developers.  This  matrix  may  continue  to 
evolve  with  successive  iterative  evaluations  of  SLEP  VELCAC  via  discussion  among  the  aforementioned 
individuals  involved  in  setting  the  baseline  for  critical  usability  attributes  and  their  acceptability  criteria. 
Regardless  of  edits  to  the  matrix,  the  attributes  presented  herein  and  their  associated  levels  of  acceptability  will 
serve  as  the  usability  standards  the  system  will  be  evaluated  against  as  it  matures. 


Table  5.  SLEP  VELCAC  Usability  Specification  Matrix. 


Attribute 

Measuring  Instrument 

Measuring  Method 

Unaccept 

-able 

Level 

Minimum 

Level 

Planned 

Level 

Best 

Case 

Level 

Time  in  errors 

•  VE  Setup 

•  Standalone 

•  ViaBMEC 

•  User  movement  in  VE 

•  Input  errors  (e.g.  clicking  on 
inaccessible  parts  of  screen) 

•  Stroking  incorrect  keyboard 
commands 

•  Disorientation  in  VE 

•  After  Action  Review 

Average  percentage 

25% 

10% 

0-10% 

• 

0% 

#  of 

subsequent 

errors 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

•  After  Action  Review 

Average  number 

>1 

1 

1 

0 

Time  to 

commence 

scenario 

•  Setup 

•  Standalone 

•  BMEC  full  setup 

•  BMEC  partial  setup 

Average  time 

TBD 

TBD 

lOmin 

5min 

Use  of  cheat 
sheet 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

•  After  Action  Review 

Average  number  in 
30mins 

>6 

6 

4 

1 

Frequency  of 
use  of  help 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

Average  number  in 
30mins 

>4 

4 

2 

0 

Number  of 
workarounds 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

•  (e.q.  turning  craft  to  change 

Average  number 

>2 

2 

1 

0 
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POV  instead  of 
manipulating  POV  via 
keyboard) 

•  Skipping  steps  due  to  system 
constraints  (e.g.  skipping  steps 
in  a  procedure  due  to  inactive 
controls/displays 

•  After  Action  Review 

Positive  vs. 

negative 

comments 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

•  After  Action  Review 

Average  percentage 

>10% 

10% 

5% 

0% 

Number  of 
comments 
expressing 
frustration 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

•  After  Action  Review 

Average  number 

>2 

2 

1 

0 

Satisfaction 

•  Setup 

•  Standalone 

•  BMEC 

•  Scenario  run 

•  After  Action  Review 

Average  rating  on  7pt 
Likert  scale  (7  is  high 
satisfaction) 

■ 

5-6 

6 

7 

Proportion  of 
users  that  find 
system  useful 

•  Perception  based  on  complete 
session 

Average  percentage 

£  60% 

70% 

90% 

100% 

Intuitiveness 

•  Memorability  of  keyboard 
commands 

•  Memorability  of  user  input 
commands 

Average  percentage 

£60% 

70% 

90% 

100% 

Utility 

•  Degree  to  which  system 
supports  user’s  task  (note: 
exact  tasks  will  be  determined 
as  system  matures) 

•  Time  it  takes  to  navigate 
between  screens/displays 
used  to  complete  a  task 

•  Switching  of  view  to  complete 
a  task 

•  Number  of 
inactive 

controls/displays 
needed  to 
complete  a  task 
(provided  task 
can  still  be 
completed) 

•  Average  number 

•  Average  Number 

2 

TBD 

2 

1 

TBD 

1 

0 

TBD 

0 

6 

0 

0 

ieU.l-yfcl.UM 

E 

•  Displays/commands  reflect 
real  world  terminology 

Average  percentage 

<90% 

90% 

95% 

100% 

Readability 

•  Percentage  of  text  critical  to 
task  completion  that  is 
readable 

Average  percentage 

100% 

100% 

100% 

100% 

TBD  =  To  be  determined 

*  VE  Setup:  steps  users  must  take  from  when  they  sit  down  at  VE  to  when  turn  computer  off  at  session  end 

*  Stand  alone:  individual  or  LCAC  crew  use 
•Via  BMEC:  using  scenario  building  tool 

*  Scenario  run:  user  interaction  with  VE  during  scenario  run 

*  User  movement  in  VE :  interaction  with  mouse/keyboard  and  VE 


•  After  Action  Review:  user  interaction  with  AAR  system 


13 


References 

Human  Factors  Design  Guide  Update  (Report  Number  DOT/FAA/CT-96/01):  A  Revision  to  Chapter  8- 
Computer  Human  Interface  Guidelines  (2001).  Springfield,  VA:  NTSI. 

Nielsen,  J.  (1993).  Usability  engineering.  Boston:  Academic  Press. 

Stanney,  K.M.,  Graeber,  D.,  &  Milham,  L.  (2002a).  Virtual  Environment  Landing  Craft  Air  Cushion  (VELCAC) 
Knowledge  Acquisition/Engineering.  VIRTE  Program  Report,  Contract  No.  N0001402C0138,  Arlington, 

VA:  Office  of  Naval  Research. 

Stanney,  K.M.,  Graeber,  D.A.,  &  Milham,  L.  (2002b).  Virtual  Environment  Service  Life  Extension  Program 
(SLEP)  Landing  Craft  Air  Cushion  (LCAC).  VIRTE  Program  Report,  Contract  No.  N0001402C0138, 
Arlington,  VA:  Office  of  Naval  Research. 

Stanney,  K.M.,  Mollaghasemi,  M.,  Reeves,  L.,  Breaux,  R.,  &  Graeber,  D.A.  (in  press).  Usability  engineering  of 
virtual  environments  (VEs):  Identifying  multiple  criteria  that  drive  effective  VE  system  design.  International 
Journal  of  Human-Computer  Studies. 

Stanney,  K.M.,  &  Reeves,  L.  (1995).  An  innovative  approach  to  usability  testing:  Facilitated  free-play. 
Proceedings  of  the  39th  Annual  Human  Factors  and  Ergonomics  Society  Meeting,  San  Diego,  CA, 
October  9-13. 

Wixon,  D.,  &  Wilson,  C.  (1997).  The  usability  engineering  framework  for  product  design  and  evaluation.  In 
M.  Helander,  T.K.  Landauer,  &  P.  Prabhu  (Eds.)  Handbook  of  Human-Computer  Interaction,  Second 
Edition.  Elsevier.  New  York:  New  York. 


14 


Appendix  A 

User  Interview  on  SLEP  VELCAC  Intended  Use  and  After  Action 

Review  (AAR) 
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While  conducting  participant  interviews  the  topics  of  SLEP  VELCAC’s  intended  use  and  After  Action  Review 
(AAR)  capabilities  were  discussed.  Presented  below  is  a  summation  of  the  feedback  on  the  aforementioned 
topics  from  the  Navigator’s  and  Craftmaster/Operator’s  perspectives.  With  respect  to  SLEP  VELCAC’s 
intended  use  the  Navigator  felt  that  the  system  would  be  ideal  for  aiding  differences  training  when  transitioning 
from  the  traditional  to  the  SLEP  LCAC.  In  particular,  demonstrating  cockpit  layout  differences,  navigating  the 
menu  structures  associated  with  the  Navigator’s  equipment,  learning  procedures  associated  with  the  UKB 
(Universal  Keyboard)  and  auxiliary  display.  To  accomplish  this,  the  Navigator  felt  SLEP  VELCAC  would  need  a 
fully  functional  UKB,  auxiliary  display,  main  display  and  trackball;  out-the-window  visuals  and  other  cockpit 
features  are  of  lesser  importance  or  not  needed.  The  Craftmaster/Operator  had  a  more  limited  input  to  the 
intended  use  of  SLEP  VELCAC,  which  is  not  surprising  given  the  minimal  differences  between  traditional  and 
SLEP  craft  for  that  crewmember.  The  Craftmaster/Operator  interviewed  felt  that  SLEP  VELCAC  would  be 
advantageous  for  gaining  an  understanding  of  the  changes  to  the  Engineer’s  position  and  capabilities  in  the 
SLEP  LCAC  (i.e.  cross  seat  training).  Other  uses  for  SLEP  VELCAC  as  a  training  system  were  not  suggested 
because  there  are  no  differences  in  flying  a  SLEP  vs.  traditional  LCAC  or  the  Craftmaster/Operator’s  immediate 
actions  in  response  to  casualties.  However,  It  was  felt  that  SLEP  VELCAC  may  prove  beneficial  for  enhancing 
crew  coordination  when  practicing  “canned”  scenarios  in  generic  environments. 

The  second  area  of  focus  during  the  interviews  was  AAR,  specifically,  discussing  what  types  of  data  need  to  be 
captured  to  create  an  effective  AAR  training  tool.  The  end  users  interviewed  were  unified  in  their  response  to 
this  topic  conveying  the  universal  benefit  of  their  input.  It  was  felt  that  differences  training  would  benefit 
immensely  from  AAR  capabilities,  in  particular,  depicting  the  user’s  path  when  navigating  the  SLEP  menu 
structures  in  contrast  to  an  optimal  path  (e.g.  fewest  steps)  to  aid  learning  of  menu  structures  and  efficient  use 
of  SLEP  upgrade  functionality.  This  capability  would  be  applicable  to  a  variety  of  scenarios  from  initializing  the 
craft  pre-flight,  to  taking  immediate  actions  as  a  result  of  a  casualty,  to  understanding  how  an  individual  arrived 
at  their  final  outcome  for  a  task  (e.g.  information  used  and  sequence  in  which  the  information  was  accessed). 
Interestingly,  this  group  of  interviewees  felt  that  some  of  the  AAR  dependent  measures  gathered  via  previous 
interviews  may  be  of  little  use.  In  particular,  the  Navigator  felt  that  measures  such  as  time  to  react  to  a  contact 
or  casualty  and  number  of  waypoints  hit  were  not  effective  data  to  present  in  AAR  due  to  the  dynamic  nature  of 
the  LCAC’s  operational  environment.  With  respect  to  using  SLEP  VELCAC  for  practicing  scenarios  in  generic 
environments,  it  was  expressed  that  data  reflecting  how  well  mission  related  criteria  were  met  (e.g.  h-hour, 
staying  within  a  swept  channel,  adherence  to  SEAOPS,  and  meeting  mission  objectives)  would  be  most  useful. 
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Appendix  B 

Response  to  Richard  Schaffer’s  Usability  Evaluation  questions 
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Here  are  some  SIM  issues  you  may  wish  to  evaluate  as  part  of  the  TEE  effort  at  IFE2.  These  would  be 
helpful  to  the  SIM  group. 

1)  Windows  are  a  problem  in  the  real  LCAC.  We  can  have  no  glass  (perfect  view),  standard  glass 
reducing  contrast),  or  cracked  glass  with  condensation.  We  have  examples  of  the  first  2. 

>  If  mission  rehearsal,  then  cracked  glass  might  be  useful  to  simulate  real  world  conditions 

>  Fog  would  be  better 

>  If  SLEP  basic  buttonology  is  the  main  training  objective,  then  standard  glass  may  be  the  best 
option 

2)  Acceptability  of  current  4  Engineer  main  screens  and  associated  functionality. 

>  Aux  screens  difficult  to  read 

>  Zoom-in  version  of  the  main  screens  are  still  difficult  to  read 

>  Functionality  isn’t  complete 

>  Screen  by  screen  review  in  upcoming  report 

3)  Value  of  "zoom-in"  keys  to  enhance  the  readability  of  main  screens. 

>  Arrows:  for  engineer  purpose  good,  need  to  improve  speed  of  access 

>  Quick  zoom  functionality  is  necessary 

4)  Review  of  rough  scaling  of  TF-40B  to  ETF-40B  engine  parameters. 

>  EGTs  would  be  850-900  at  95%  N2 

5)  Review  of  Fuel  Manifold  failure  procedures  and  simulator  responses. 

>  Update  available  (Chris  also  has  partial  data  on  this) 

6)  Utility  of  low  cost  HMDs  (2  available). 

>  No  for  Navigator  and  Engineer,  yes  for  Craftmaster 

7)  Utility  of  AARS  playback  of  displays. 

>  Get  more  focused  data  than  a  playback  (see  ACU5  report) 
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Appendix  C 

Usability  Attribute  Table 
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Table  C.  Usability  attribute  table. 
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In  LCAC,  background  noise  is  •  May  be  interested  in  the  speed  and 
very  loud,  need  to  consider  if  power  settings’  impact  on  beach 

background  should  be  landing 

simulated  _ 


Appendix  D 

User  Profile  Questionnaire 
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We  would  be  most  appreciative  if  you  would  complete  this  questionnaire  so  we  may  better  understand  you  as  a 
user.  Thank  you  for  your  time  and  your  participation  is  greatly  appreciated. 

1 .  Your  job  title  is  (e.g.  Craftmaster,  Navigator,  Engineer): - 

2.  How  many  hours  have  you  flown  on  NDI  equipped  LCACs? _ 

3.  How  many  hours  have  you  flown  on  non-NDI  equipped  LCACs? - 

4.  How  many  hours  have  you  flown  on  the  SLEP  LCAC? - 

5.  Are  you  an  instructor  on  the  SLEP  LCAC? _ 

If  so,  briefly  describe  the  course  content  (e.g.  highlighting  differences  between  traditional  and  SLEP  LCAC, 
etc.) _ not  designed  yet 


6.  Describe  the  current  level  of  automation  of  your  job  while  in  the  SLEP  LCAC: 

_ None  (There  is  no  automation  of  my  job  in  the  SLEP  LCAC) 

_ Low  (Use  SLEP  LCAC  controls  and  displays  1-30%  of  the  time  in-flight) 

_ Medium  (Use  SLEP  LCAC  controls  and  displays  31-70%  of  the  time  in-flight) 

_ High  (Use  SLEP  LCAC  controls  and  displays  71  -1 00%  of  the  time  in-flight) 

7.  In  general  how  do  you  feel  about  working  with  computers? 

_ I  don’t  like  working  with  computers. 

_ I  have  no  strong  like  or  dislike  for  working  with  computers. 

_ I  like  working  with  computers. 

_ Other  (please  explain)  _ _ _ 
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8.  How  have  computers  affected  your  job? 


_ Automation  in  the  SLEP  LCAC  has  made  my  job  easier. 

_ Automation  in  the  SLEP  LCAC  has  not  affected  my  job  in  any  particular  way. 

_ Automation  in  the  SLEP  LCAC  has  made  my  job  more  difficult. 

_ Other  (please  explain) _ _ _ _ _ 

9.  Is  the  amount  of  time  it  takes  to  leam  new  computer  applications  usually  worth  it? 

_ Yes,  it  pays  off  because  computer  systems  usually  help  me  do  my  job  better  or  faster. 

_ Sometimes,  it  pays  off,  and  sometimes  it  doesn’t. 

_ No,  computer  systems  are  usually  not  useful  enough  to  justify  the  training  time. 

_ Other  (please  explain) _ _ 

10.  Do  you  enjoy  learning  how  to  use  new  computer  applications? 

_ Yes,  it’s  usually  challenging  and  interesting. 

_ Sometimes,  depending  on  the  application. 

_ No,  it's  usually  tedious  and  frustrating. 

_ Other  (please  explain) _ _ _ _ _ 

1 1 .  In  general,  are  you  interested  in  computers? 

_ I  am  not  interested  in  computers  and  would  avoid  using  them  if  I  could. 

_ I  am  interested  in  computers  but  only  as  a  means  to  help  me  do  my  job  better  and 

faster. 

_ I  am  interested  in  computers  in  general,  and  I  enjoy  using  them. 

_ Other  (please  explain) _ _ 

12.  How  many  years  since  you  became  “certified”  in  your  current  LCAC  crewmember  position: _ _ 


13.  What  is  your  highest  academic  degree? 


_ No  degrees 

_ High  school  degree 

_ Trade  or  vocational  school  degree  (beyond  the  high  school  level) 

_ College  degree  (for  example,  B.A.,  B.S.,  Associate) 

_ Graduate  degree  (for  example,  M.A.,  M.S.,  Ph.D.,  Ed.D.,  M.D.,  R.N.) 

_ Other  (please  explain) _ _ 

14.  How  would  you  describe  your  skill  level  as  a  “certified"  LCAC  crewmember  in  your  current  job  title? 

_ Novice 

_ Experienced 

_ Expert 

_ Other  (please  explain)  ___ - 

1 5.  How  would  you  describe  your  skill  level  as  a  “certified”  SLEP  LCAC  crewmember  in  your  current  job  title? 

_ Novice 

_ Experienced 

_ Expert 

_ Other  (please  explain) _ _ _ _ _ 

16.  How  would  you  describe  your  general  level  of  computer  experience? 

_ None  (I  have  never  used  any  computer  applications). 

_ Low  (I  have  used  only  1  or  2  computer  applications). 

_ Moderately  Low  (I  have  learned  and  used  between  3  and  10  different  computer  applications). 

_ Moderately  High  (I  have  learned  and  used  more  than  10  different  computer  applications  but  have  no 

programming  skills). 

_ High  (I  have  used  many  different  computer  applications  and  have  some  programming  skills). 

_ Other  (please  explain) _ _ _ 
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17.  List  all  training  systems  you  have  used  and  the  amount  of  experience  in  years  or  months  that  you  have 
used  each  system. 

Training  System  Experience  (specify  years  or  months) 


18.  In  general,  the  training  systems  you  have  used  have  been: 

_ Completely  effective  in  training  critical  skills  and  job  practices. 

_ Somewhat  effective  in  training  critical  skills  and  job  practices. 

_ Of  little  worth  in  training  critical  skills  and  job  practices. 

_ Other  (please  explain) _ _ 

19.  In  general,  the  training  systems  you  have  used  have  been: 

_ Easy  to  use. 

_ Somewhat  easy  to  use. 

_ Somewhat  difficult  to  use. 

_ Difficult  to  use. 

_ Other  (please  explain) _ 

20.  Are  you  (check  one)?  _  Male  _  Female 

21 .  How  old  are  you  (check  one)? 

_  18-25 

_  26-40 

_  41-55 

_  over  55 
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22.  Do  you  wear  glasses  or  contact  lenses  (check  one)? 


No 

Yes  (Please  check  your  vision  problem  and  correction  method) 

_  Nearsighted 

_  Farsighted 

_  Astigmatism 


_  Glasses 

_  Bifocals 

_  Contact  lenses 

Are  you  color  blind  (check  one)? 

_  No  _  Yes 

23.  Do  you  have  accurate  depth  perception?  (T est:  Extend  your  arm  straight-out  in  front  of  you  at  shoulder 
height.  Point  your  index  finger  to  the  ceiling  and  reference  it  to  an  object  on  the  wall.  Close  one  eye,  then 
close  the  other.  As  you  alternate  eyes,  does  the  finger  move  its  position  relative  to  the  reference  object?) 

_  Yes 

_  No 


24.  Do  you  have  any  physical  conditions  other  than  vision  deficiencies  that  computer  technology  would  need  to 
accommodate  or  support  (e.g.,  hard  of  hearing,  arthritis  in  hands,  wheelchair)? 

_  No 

_  Yes  (Please  describe) _ _ _ 
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Appendix  E 


Update  on  Casualty  Mitigation  Procedures  for  Loss  of  APU 


32 


Engineering  Casualty:  Loss  of  APU 
Break  down  of  the  task 

Dealing  with  the  loss  of  APU:  loss  of  port  primary  pump 
>  Step  1 :  The  Engineer  would  first  zoom  into  the  aux  screen  to  read  the  caution  (note:  the  secondary 
pump  caution  wouldn’t  appear  immediately,  around  a  30  second  delay) 

o  In  the  aux  display  the  port  pump  primary  should  be  switched  to  secondary  (verify  that  port 
secondary  pump  came  on) 

o  The  Engineer  would  then  look  at  the  main  page  to  verify  that  port  manifold  pressure  is  steady 
(will  be  steady  for  about  30  seconds  or  so,  then  if  the  secondary  fails,  he  will  start  to  see  it 
continue  to  decrease) 

o  If  the  secondary  fails,  he  will  see  the  caution  on  the  aux  screen  for  the  secondary 
o  He  will  go  to  ME  FEED  page  (if  it  shows  steady  pressure,  then  you’re  done  -  monitor  on  the 
main  page  or  the  ME/  feed/  transfer  page) 

■  If  the  pressure  is  not  steady,  then  he  will  open  crossover 

■  Then,  he  will  verify  on  the  ME  FEED  page  that  port  manifold  pressure  is  steady,  if  still 
decreasing,  he  is  going  to  close  the  cross  over  valve  [in  UKB] 

o  If  the  port  manifold  pressure  is  still  decreasing,  the  Engineer  will  go  to  the  Main  in  UKB  and 
take  the  following  steps: 
o  Go  to  main  engine  UKB  page 

■  He  will  set  the  engines  to  idle:  problem  is  that  the  3  row  aren’t  working,  would 
hit  them  for  1  and  2) 

•  Bring  the  engines  to  idle  (UKB:  M/E  1  and  2  would  say  IDLE,  hit  them 
both) 

•  He  would  then  stop  them  on  the  same  page  (UKB:  M/E  1  and  2  hit  stop) 
o  Go  back  to  UKB:  Main 

o  Go  to  UKB:  APU  ELEC 

■  He  would  turn  Gen  2  off 

■  He  would  then  reach  up  and  secure  APU  #2 
o  Go  to  UKB:  main 

o  Go  to  UKB:  ME  FEED 

•  Close  port  tank  valve: 

o  UKB:  Hit  p.  tank  closed 

•  Forward  feed  power  off 

o  UKB:  hit  fwd  fd  off 

o  Finally,  he  would  bring  craft  hull  borne  to  investigate 
•  Go  to  UKB  main 

■  Close  cushion  vanes 

•  P.  vane  close 

•  S.  vane  close  (note:  shows  the  actual  function) 
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Appendix  F 

Usability  concerns  with  general  usability  and  virtual  environment 

heuristics  and  their  severity 
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Heuristic  Evaluation 

Presented  below  are  the  heuristic  evaluation  findings  for  both  the  general  and  virtual  environment  heuristics. 
For  each  heuristic  a  table  is  provided  that  summarizes  usability  concerns  for  that  heuristic,  the  severity  of  the 
concern,  and  whether  or  not  participants  noted  the  concern  as  an  issue  without  prompting  during  user  testing. 

General  Usability  Heuristics 


Simple  and  Natural  Presentation 

Table  F-1 .  Concerns  with  the  simple  and  natural  presentation  heuristic  and  their  severity 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

Usina  the  mouse  to  interact  with  the  UKB  is  not  natural 

Low 

Yes 

Usina  the  mouse  to  interact  with  other  switches,  knobs,  dials  is  not  natural 

Low 

Not  examined 

Rotating  point  of  view  up  with  the  “S”  key  and  down  with  the  “W”  key  is 
counter  intuitive 

Med 

Yes 

Unnatural  to  zoom  via  repeatedly  depressing  keys 

Med 

Yes 

Chanqinq  point  of  view  via  repeatedly  depressing  keys  is  unnatural 

Yes 

Unnatural  flow  of  information  gathering  and  integration  resulting  from 
having  to  zoom  in  on  the  main  or  auxiliary  display,  which  excludes 
supplemental  information  provided  on  the  other  screen 

High 

Yes 

Users  did  not  realize  that  the  upper  panels  for  each  crewmember  position 
were  represented 

Low 

Yes 

Users  suggested  horizon  presentation  wasn’t  correct,  too  much  water,  not 
enough  sky 

Low 

Yes 

There  was  no  collision  detection  for  the  craft  when  flying  overland 

Low 

Not  examined 

Alarms  on  engineer’s  auxiliary  page  flash  in  the  actual  craft  when  they 
have  not  been  acknowledged,  but  they  do  not  in  SLEP  VELCAC 

Med 

Yes 

Inactive  keys  are  causing  confusion  because  they  are  in  the  incorrect  non¬ 
default  condition 

High 

Yes 

Speak  the  User’s  Language 

Table  F-2.  Concerns  with  the  speak  the  user’s  language  heuristic  and  their  severity 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

User’s  language  is  not  utilized  to  its  fullest  extent  possible  in  the  “cheat 
sheet” 

Low 

Yes 

Minimize  Memory  Load 

Table  F-3.  Concerns  with  the  minimize  memory  load  heuristic  and  their  severity 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

Memory  taxed  with  having  to  learn  and  remember  keyboard  commands 

Med 

Yes 

Memory  load  is  taxed  by  necessity  to  zoom  in  on  the  main  or  auxiliary 
display  to  read  its  information,  which  hinders  ability  to  see  the  other  display 

High 

Yes 

Labeling  of  displays  is  hard  to  read  thus  requiring  the  user  to  recall  from 
memory  functionality  accessed  via  the  labeled  controls 

High 

Yes 
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Consistency 

Table  F-4.  Concerns  with  the  consistency  heuristic  and  their  severity 


Usability  Concerns 


Inconsistent  function  for  the  “1”  key;  first  press  shows  Navigator’s  main 
display,  subsequent  press  shows  last  point  of  view 


Inconsistent  functioning  of  the  “ack"  key  to  acknowledge  alerts  and  alarms 


Feedback 

Table  F-5.  Concerns  with  the  feedback  heuristic  and  their  severity 


Usability  Concerns 


Visuals  do  not  give  any  sense  of  craft  speed _ 


No  collision  detection  over  land  _  _ 


System  locked  up  without  telling  the  user  why  it  locked  u 


No  indication  of  bow  thruster  status  (stow  v.  operate)  or  direction  (forward 
v.  reverse)  without  viewing  the  synthetic  HUD,  which  does  not  exist  in  the 
craft  _ _ _ 

Clear  Exits 

Table  F-6.  Concerns  with  the  clear  exits  heuristic  and  their  severity 


Usability  Concerns 


No  ability  to  go  back  to  a  particular  part  of  a  scenario  (can  only  restart 


No  ability  to  “undo”  to  avoid  catastrophic  errors _ _ 


Severity 


Severity  Validated  by 
User  Testing? 


Med  Yes 


Med  Yes 


Yes 


Shortcuts 


Usability  Concerns 

Severity 

vnmmmm 

■BSEmiffljMl 

No  ability  to  choose  where  along  a  route  or  task  completion  to  start  a 
scenario 

Med 

Not  examined 

Errors,  Error  Handling,  Error  Prevention 


Usability  Concerns 

Severity 

BUliM 

When  system  locks  up  there  is  no  indication  of  why  or  when  via  error 
messages 

Med 

Not  examined 

No  ability  to  undo  errors,  thereby  avoiding  more  critical  errors 

Yes  | 

Repeated  system  lockups  required  rebooting  of  the  system 

■a 

Yes 

When  transitioning  from  surfzone  to  land  the  Craftmaster/Operator’s 
heading  indicator  malfunctioned 

High 

Not  examined 

When  flying  over  land  the  engineer’s  controls  became  inactive 

■m 

Yes 

Zoom  in  on  engineer’s  main  display  (i.e.  the  4  button)  was  not  completely 
functional,  it  needed  to  be  depressed  numerous  times  (in  some  examples 
15  times)  before  performing  it’s  function 

High 

Yes 
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Help 

Table  F-9.  Concerns  with  the  help  heuristic  and  their  severity 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

No  on-line  help  available 

■m 

Yes 

Current  Daoer  based  help  does  not  provide  timely  access  mid-scenario 

■m 

Yes 

Virtual  Environment  Specific  Heuristics 

Of  the  virtual  environment  specific  heuristics  discussed  above  in  Table  3,  only  the  interaction,  navigation,  and 
visual  heuristics  were  found  to  have  violations  and  thus  the  remaining  heuristics  are  not  listed  below. 


Interaction 

Table  F-10.  Concerns  with  the  interaction  heuristic  and  their  severity 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

Interaction  with  SLEP  menu  screens  limited  to  UKB 

Low 

Yes 

Engineering  station  does  not  allow  interaction  with  buttons  on  overhead 
console 

Low 

Not  examined 

Navigation 

Table  F-1 1 .  Concerns  with  the  navigation  heuristic  and  their  severity 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

External  cues  (waves,  land,  etc.)  do  not  afford  estimation  of  craft  speed 

Low 

Not  examined 

Lack  of  collision  detection  allows  user  to  zoom  point  of  view  outside  of 
cockpit 

Low 

Not  examined 

Zooming  should  be  scaled  to  human  movements  or  driven  by  tracking 
head  movements  instead  of  “jumping”  to  an  expanded  view  of  a  particular 
screen 

Med 

Yes 

Movement  within  the  VE  is  cumbersome  due  to  the  need  for  repetitive 
key  strokes  to  change  point  of  view 

High 

Yes 

Visual 

Table  F-1 2.  Concerns  with  the  visual  heuristic 


Usability  Concerns 

Severity 

Validated  by 
User  Testing? 

Text  on  main  and  auxiliary  displays  is  illegible  when  zoomed  out  and 
occasionally  illegible  when  zoomed  in 

High 

Yes 

When  transitioning  from  surfzone  to  land  a  greenish  brown  bar  fills  a 
majority  of  the  screen 

High 

Not  examined 

